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REAL TIME, PROGRAMMABLE CHROMA KEYEvJG 
WITH SHADOW GENERATION 



Field of the Invention 
5 This invention relates to presentation of television signals, using 

chroma keying to combine and present images. 
Background of the Invention 

In formation of a television signal, several distinct views may be 
available for use at any time. In many instances, the broadcast controller 

10 may choose to present a single view. However, in some instances, it may 
be preferable to merge, combine and/or modify images from two or more 
views, using a process caUed chroma keying. A chroma key specifies, 
pixel by pixel, whether a portion of a foreground view or of a background 
view will be displayed. In chroma keying, a foreground portion of a first 

15 view or source may be combined with a background portion of a second 
view or source, by selectively replacing television signal data from the 
first source with data from the second source. When the foreground data 
from a particular source is captured, a selected key color is usually chosen 
for the (initial) background. A chroma key algorithm identifies that key 

20 color within the foreground data to determine where, or whether, to 
combine or merge the background data. 

Several difficulties occur in implementation of chroma keying. The 
key color used as a backdrop may not have uniform luminance or chroma, 
due to physical defects in the display screen, imperfect lighting, the 

25 presence of other (nonuniform) lighting sources or reflections or shadows, 
and for other similar reasons. A relatively transparent foreground object 
win often exhibit the chroma key color and thus be undesirably replaced 
with a portion of the background image. The edges of a foreground image 
may manifest fine detail, such as wisps of hair, smoke, etc. that are easily 

30 lost using many chroma key algorithms. Color spill or infection from the 
chroma key color is often present adjacent to the edges (sharp or diffuse) 
of a foreground object and may arise from reflected key color light from 
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the backdrop, blurring of an image due to local defocusing of the camera, 
motion of the camera and object relative to each other, video filtering, etc. 

Originally, the chroma key was a "hard" key, within which only the 
foreground image or only the background image was displayed at each 
5 pixel. A "soft" chroma key has become available more recently, in which 
each pixel may include a first fraction of the foreground image and a 
second fraction of the background image. Use of soft keying allows a 
blending at an image edge and may provide a more natural transition 
between images. 

10 Figure 1 is a block diagram of a conventional chroma key system 

1 1 . A foreground image and a background image are received on signal 
lines 13 and 14, respectively, at a foreground suppression module 15 and 
at a background suppression module 17. A foreground key generator 19 
and a background key generator 21 provide a foreground key and a 

15 background key for the foreground suppression module 15 and the 
background suppression module 17, respectively. The foreground 
suppression module 15 and the background suppression module 17 provide 
partially (or fully) suppressed foreground and background images that are 
received by an image sum module 23. The result is a chroma keyed image 

20 that may combine and issue the foreground and background images in a 
composite video image according to the particular keying algorithm 
adopted. 

Figure 2 is a block diagram of an existing alpha mixer system 31. A 
foreground video signal FG and alpha signal a (0 < a < 1) are received on 

25 input signal lines 33 and 35, respectively, and are multiplied to form a 
first product, a-FG, m a first multiplier module 37. A background video 
signal BG is received on a third input signal line 39, an a-complement, 
1 - cx, is formed in a complement module 41, and a second product, (1 - 
a)*BG, is formed m a second multiplier module 43. A sum of the first 

30 product and the second product is formed in a sum module 45, and the 
resulting mixed video signal, cc-FG + (1 - a)-BG, is issued as an output 
signal on an output line 47. 
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What is needed is a soft chroma keying approach in which one or 
more foreground images can be combined with one or more shadowed 
background images in a programmable approach that allows the relative 
strength of each image to vary, pixel by pixel, with time across the display 
5 screen. Preferably, the sum of the strengths of the contributing images 
should not be required to add to 1.0. Preferably, the distribution of 
chroma key values over a color plane ( a "key map") should be expressible 
as a mapping between selected foreground location on the color plane and 
corresponding n-tuple representing the chroma key values for each pixel. 
10 Summary of the Invention 

These needs are met by tihe invention, which provides a simplified 
approach for generation of chroma key, alpha mix value and shadow 
generation that reduces memory requirements, substantially reduces the 
number of multiplication and division operations for the numerical 
15 computations, preserves real time analysis and graphical display, and 
provides natural, acceptable visual images. 

In one embodiment, s sum of a suppressed foreground image with 
weight a , with one or more selected foreground colors suppressed, and a 
background image with weight 1 - a(0<a<l)is formed, where a may 
20 vary from pixel to pixel and with time. In another embodiment, a shadow 
from a selected foreground image is impressed on selected pixels of a 
background image. In another embodiment, foreground suppression and 
shadowing are combined. Optionally the new system may be formed by 
retrofitting, using an existing a-mixer, or the system can be constructed 
25 from components. 

Brief Description of the Drawings 

Figures 1 and 2 are block diagrams of a conventional chroma key 
system and an existing alpha key mixer system, respectively. 

Figure 3 is a block diagram of a chroma key system using an 
30 existing alpha mixer according to the invention. 

Figures 4A(prior art) and 4B are block diagrams of systems for 
shadow generation that scales background luminance. 
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Figure 5 is a block diagram of a system for combined foreground 
suppression and shadow generation using an existing alpha mixer and 
associated alpha values according to the invention. 

Figures 6 and 7 are an example of a chroma key map that can be 
5 prescribed according to the invention. 

Figure 8 is a graphical illustration of key value clipping and 
rescaling used in the invention. 

Figure 9 is a schematic view of a computer system. 
Description of Best Mode of the Invention 
10 In a conventional approach to alpha mixing, as illustrated in Figure 

2, a resulting or composite image is a convex sum of a foreground image, 
with associated weighting a, and a background image, with weighting 1-a. 

Figure 3 illustrates a system 51 that allows use of combinations of 
foreground and background images, where the sum of the weighting 
15 coefficients may be greater than, equal to or less than 1 for any pixel. A 
foreground video image is received on a first input signal line 52, at a 
foreground suppression module 53 and at a foreground key generation 
module 55. The foreground key generation module 55 issues a soft 
foreground key signal on a first intermediate signal line 57 that is received 
20 by the foreground suppression module 53 and by a foreground key to a 
module 59. 

The foreground suppression module 53 receives the two input 
signals and issues a suppressed foreground video signal SuFG on a second 
intermediate signal line 61 that is received by a foreground image product 

25 module 65 that is part of a standard alpha mixer sub-system 63, similar to 
the sub-system shown in Figure 2. For any pixel or group of pixels, the 
suppressed foreground signal SuFG received on the second intermediate 
signal line 61 may have an associated color vector amplitude for any pixel 
that is different from the distance from the color plane origin to the 

30 location of the original foreground color in the color plane. The 

foreground image product module 65 also receives an alpha value signal, 
a = a(FG), issued by the foreground key to a module 59 on a third 
intermediate signal line 67, and forms and issues a first product, a*SuFG, 
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of the a value and the suppressed foreground video signal on a signal line 
66 that is received by a first input terminal of a sum module 77. 

A background video signal BG is received on a second input signal 
line 71 by a background image product module 73. The a-key signal 
5 issued by the a-key module 59 is also received by an a-complement 
module 69, which forms and issues an a-complement value, 1-a, on a 
signal line 70. The a-complement signal is also received by the 
background image product module 73. The background image product 
module 73 forms and issues a second product, (l-a)-BG, which is received 
10 by the sum module 75. The sum module 75 forms and issues a composite 
simi signal, a-SuFG + (l-a)-BG, on an output signal line 77. An existing 
alpha mixer sub-system 63 may be retrofitted with the remainder of the 
components shown in Figure 3, or the entire system 51 can be constructed 
initially. 

15 More generally, the sum module 75 may form and issue a 

suppressed foreground composite image signal, a-SuFG -i- (l-a')-BG, 
where a and a' are selected real numbers lying in a range between 0 and 
1. Where the suppressed foreground image intensity is to be limited by the 
background image intensity for a given pixel, the constraint 0 < a' < a < 

20 1 can be imposed. Where the background image intensity is to be limited 
by the suppressed foreground image intensity for a given pixel, the 
constraint 0 < a < a' < 1 can be imposed. 

A foregroimd key and associated key map can vary the suppressed 
foreground video signal SuFG from 0 percent to 100 percent in selected 

25 increments, for any pixel. Preferably, this suppression key is generated 
automatically. 

Foreground suppression can eliminate all evidence of a selected key 
color from the foreground video image, before the (modified) foreground 
video image is mixed with the background video image. Without 
30 application of foreground suppression, a region of chroma spill on the 
foreground image wiU become very noticeable in a composite image. 
Normally, insufficient information is available to correctly replace all of 
the key color spill with the true colors that would be available in a neutral 
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background. One goal of foreground suppression is to create a natural 
looking foreground image that does not contain the key color. The 
mapping between the foreground points of the color plane and the post- 
suppression points on the color plane is central to the quality of the 
5 suppression key. On a key generation map, every chroma point has an 
associated key value. On the suppression map, by contrast, every chroma 
point has an associated vector (or mapping) that points to or indicates the 
resulting chroma point after suppression is implemented. 

Figure 4 A illustrates a conventional shadow generation system 81 A 

10 that can be used to generate a background image having a "shadow" that is 
determined with reference to a foreground image. A foreground video 
image is received on a first input signal line 83 at a shadow control 
module 85A and at a shadow generation module 87 A. The shadow control 
module 85A receives a foreground key signal on a second input signal line 

15 89, analyzes the foreground key signal, and issues a pixel-by-pixel binary 
or on/off shadow control signal that is received by the shadow generation 
module 87A. The shadow generation module 87A forms and issues a 
foreground shadow signal FGSh, in which each selected foreground pixel 
appears in a shadow format, on an intermediate signal line 91, which is 

20 received by a shadow multiplier module 95 A. A background luminance 
signal BGL is also received on a third input signal line 93 at the shadow 
multiplier module 95A, and a modified background luminance signal 
MBGL, with the foreground shadow imposed on selected pixels, is formed 
and issued on an output signal line 97 A. 

25 Figure 4B illustrates a shadow generation system 8 IB that can be 

used to generate a background image having a shadow, determined with 
reference to a foreground image, according to an embodiment of the 
invention. The shadow control module 85B and shadow generation module 
87B in Figure 4B are similar to the corresponding components shown in 

30 Figure 4A. The conventional multiplier module 95A in Figure 4A is 
replaced in Figure 4B by a combiner module 95B that uses summation 
rather than multiplication, together with the background luminance signal. 
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to form and issue a modified background luminance signal MGBL' with 
shadowing on the output signal line 97B. 

One method of impressing a foreground shadow onto a background 
image is to replace the "shadowed" portions of the background image by a 
5 sum, p-FGSh + (l-p')-BG, of the background image EG and the 

foreground shadow image FGSh, where p and P' are two selected real 
numbers satisfying 0 < p < p' < 1. This approach ensures that the portion 
of the background image BG that is overlain by the foreground shadow 
image FGSh has a shadowed background image intensity that is limited by 

10 the associated foreground image shadow intensity (0 < 1-P' < 1-p). As a 
first altemative, one can select the numbers b and b' to satisfy 0 < p' < P 
< 1 so that the associated foreground image shadow intensity is limited by 
the background image intensity. As a second altemative, one can choose P 
= p' for the selective shadowing of background by foreground. The value 

15 of p and/or P' may be chosen independently of, or to reflect a numerical 
relationship to, the alpha values a and/or a' used in the discussion of 
Figure 3. 

Where the key color background is approximately uniformly lit, 
shadows from objects in the foreground may be visible. When these 

20 patterns of luminance attenuation are transferred to the background image, 
it should appear that the foreground objects have cast their shadows on the 
background images. Where the luminance attenuation is done correctly, 
the realism of the composite image is enhanced. This same process can be 
used for translucent and/or transparent images, for example, where a 

25 background image is viewed through a foreground image such as fine hair 
or mist or a partially transparent window pane. Preferably, shadow 
generation is performed in real time, as each composite image is formed. 

Figure 5 is a block diagram of a system 101 for shadow generation 
and the chroma key algorithm discussed in the preceding, using an existing 

30 or specially constructed alpha mixer sub-system, to provide a composite 
image with suppressed foreground and foreground image shadowing of 
selected background objects. A foreground video signal is received on a 
first input signal line 102 at a foreground suppression module 103, at a 
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foreground key generation module 105, at a shadow control module 107 
and at a shadow generation module 109. The foreground key generation 
module 105 generates and issues a foreground key signal FGK on a first 
intermediate signal line 111, and the signal FGK is received at the 
5 foreground suppression module 103, at the shadow control module 107 
and at a foreground key to alpha module 113. The shadow control module 
107 generates and issues a foreground shadow control signal FGSh, on a 
second intermediate signal line 115, that is received by the shadow 
generation module 109. The shadow generation module 109 generates and 

10 issues a shadow generation signal ShG on a third intermediate signal line 
117, which signal is received at a combiner module 121. 

The combiner module 121 takes account of the shadowing of 
selected pixels in a shadowed background image by the foreground image 
for the corresponding pixels, in generating a modified alpha key, denoted 

15 as p; where no shadowing is present, p = a for this portion of a composite 
image; where shadowing is present, the formalism developed in the 
discussion of Figure 4B is applied, and the shadowed portion of the 
background image BG is replaced by a modified background luminance 
signal, MBGL' = p-FGSh + (l-p')-BG, where p and p' are selected real 

20 numbers in the range [0,1]. 

The foreground key to alpha module 113 generates and issues a 
foreground key to alpha signal FGKA on a fourth intermediate signal line 
119 that is also received by the combiner module 121. The combiner 
module 121 generates and issues an alpha signal pair (a,a') and/or a 

25 modified alpha signal pair (p,p'), with pair values that take account of the 
shadow, if any, to be generated for each pixel and the composite image of 
foreground image and backgroimd image that is ultimately to be formed, 
hi some simple situations, the pair (a,a') or (P,p') reduces to a single 
value, such as a or P, that is received and processed by an alpha mixer 

30 module 125. 

The foreground suppression module 103 generates and issues a 
suppressed foreground signal SuFG on a fifth intermediate signal line 123 
that is received by an alpha mixer module 125. The alpha mixer module 
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125 also receives a background video signal on a second input signal line 
127 and forms and issues a composite signal, a-SuFG + (l-a')*BG (no 
shadow) or a-SuFG + (l-a')-MBGL (with shadowing), an output signal 
line 129. 

5 Alternatively, the combiner module 121 and the alpha mixer module 

125 can work cooperatively to provide a compensated alpha value, a" with 
0 < a" < a, and form and issue a composite signal a"-SuFG + (l-a")-BG 
on the output signal line 129. This alternative approach is appropriate, for 
example, where an existing alpha mixer is to be reused, or retrofitted for 

10 use with, the present invention. The alpha mixer module 125 can be 

reused, with many or all of the other components shown in Figure 5 being 
retrofitted, or the system 101 shown in Figure 5 can be constructed from 
available and newly designed components. 

One advantage of the invention is the flexibility of the key map, 

15 which is programmable pixel-by-pixel if desired. Any key color can be 
chosen and any distribution of the key map can be used. If a user 
determines that a different contour and/or different color range for the 
key map is more desirable, the system can be reprogrammed to 
accommodate this change. The key map may, for example, be 

20 programmed for a specific distribution of colors within a particular 
sequence of video images. 

The invention minimizes much of the hardware complexity of 
chroma keymg, by eliminating many of the multiply and divide operations 
associated with the chroma key calculations. The invention allows re-use 

25 of an already- available alpha mixer. By incorporating the requirement that 
the chroma key map is non-zero only within a reasonably bounded region, 
which is relocatable, the memory required to store the key map is reduced 
substantially. By incorporating the assumption that the chroma key values 
will be continuous and will not have associated high spatial frequencies 

30 within the bounded region, aggressive sub-sampling of the bounded region 
further reduces the memory requirements. The foreground suppression 
module (103 in Figure 5) re-uses the key map data, rather than requiring 
allocation of additional memory for storing the suppression vector map. 
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The key map data are also used to construct the alpha values for the video 
alpha mixer. The invention generates alpha values from the chroma key 
signal and independently generates color vectors corresponding to the 
chroma key signal, which offers greater control of the level of foreground 
5 suppression, independent of alpha mixer operation. 

Shadow generation according to the invention is a substantial 
departure from conventional shadow generation. According to the 
invention, a shadow that is visible on the composite image is related to, but 
is not necessarily proportional to, the shadow on the underlying 

10 foreground image. A luminance difference value, rather than a luminance 
ratio, is used in the shadow algorithm. This new approach maintains subtle 
shadings but reduces the complexity of the computations. The strength of a 
shadow in the conventional approach and in the invention are generally not 
the same, but the shadows appear natural in both approaches. 

15 The shadow information is incorporated into the alpha value, not 

into the background luminance or into the final composite image as is 
usually done in conventional approaches. A shadow is generated on the 
final composite image by modifying the alpha value to mix in more black 
(and/or remove more of the foreground color). This modification 

20 eliminates many of the multiply operations that are normally required by a 
conventional shadow algorithm, allowing use of (only) the alpha mixer. 

Figure 6 is a graphical view of representative Cb versus Cr color 
values, showing a result of generating a key map in a bounded region of 
this color plane. For this example, a central region C (preferably convex) 

25 is black, indicating that a (foreground} pixel color within C is completely 
replaced by a corresponding (background) pixel color having its own 
color distribution. An annular region T immediately surrounding the 
center C is gray, indicating one or more transition regions where a pixel 
having a color within T is partly, but not completely, replaced by a 

30 corresponding background pixel. The remainder of the color plane in 
Figure 6 is white, colorless or unmodified, with no full or partial 
replacement of a foreground pixel color by a corresponding background 
pixel color. The center region C and the transition region T need not be 
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circular and/or annular and may have any reasonable shapes. Figure 7 is a 
graphical view of a typical key, representing a pixel replacement index, 
which is 1.0 within the region C, for a color radius r < C^, decreases 

monotonically within the region T, and is 0.0 for a color radius greater 
5 than second selected radius Tg (> Cq). 

The foreground key generation module (55 in Figure 3, 105 in 
Figure 5) produces a foreground key from the chrominance and 
luminance information for each pixel of the foreground video, normally 
using a lookup table. However, for a ten-bit component, this foreground 

10 key generation would require provision of about 2^^ (~ 10^) memory 
addresses for a lookup table. If the luminance value is temporarily 
ignored, the required lookup table memory is reduced to about 2^^. By 
confining the 2^^ nonzero values of a foreground suppression key value to 
within a rectangular bounding box containing the chroma key color(s), as 

15 suggested in Figure 6, the lookup table memory requirements may be 
further reduced to about 2^7 addresses. By further assuming that the 
distribution of key values within the bounding box is continuous and 
involves only gradual changes in key values (no high spatial frequencies), 
the lookup table memory need only store down-sampled values, fewer in 

20 number than would otherwise be required. The key generation module 
determines a pixel's key value by (1) determining if the key is within a 
bounding box, (2) if no, the key is set to 100 percent foreground, and (3) 
if yes, the key value is interpolated based upon the "nearest" sample values 
stored in memory. 

25 A foreground suppression module (53 in Figure 3, 103 in Figure 5) 

removes from the video image the chroma key color and partly removes 
any transition colors between the key color and the colors of the true 
image. This color removal is implemented by using the color key signal as 
an indicator of how much of the key color has been mixed to form the 

30 present key pixel value. For the key color itself, indicated by 0 percent 
foreground, the foreground suppression module will attempt to subtract a 
color vector from the key vector to produce a neutral color, such as black, 
gray or white. If, for example, (Cb, Cr) = (128, 128) is taken as the 
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origin, the vector to be subtracted will extend from (128, 128) to a target 
key color (Cb(key), Cr(key)). For a key signal value that indicates 100 
percent foreground, for example, no subtraction is performed. For an 
intermediate key vector value, a vector with a proportionally reduced 
5 length is subtracted. 

A foreground key to alpha module (59 in Figure 3, 113 in Figure 5) 
modifies the foreground key so that the key can be used to define an alpha 
value to drive the alpha mixer module (63 in Figure 3, 125 in Figure 5). 
If an unmodified key is used to define the alpha value, some areas of key 

10 color transition, such as color spill or key color reflections from another 
surface, may become translucent, and the background video image may 
appear in inappropriate places. Optionally, the foreground key to alpha 
module (1) clips a key value K (0 < K < 1) below a lower threshold value 
Kjnin and/or clips the key value above an upper threshold value Kj^^x 

15 does not affect the key value between these two threshold values, as 

illustrated in Figure 8, and/or (2) lowers and rescales the resulting mid- 
range of key values linearly to cover the full span of alpha values. A 
suitable set of defining equations for these two processes is 

al = f(a) = 0.5 {la - a^i^\ - la - aj^ax' + ("min + "max)}' d) 

20 a2 = g(al) = (al - ajnin)/(ccmax " «min)- (2) 

The modified key value, a = Kl or a = K2, is the result of applying one 
or both of the two processes indicated in (1) and (2). Optionally, one of 
the processes in (1) and (2) can be apphed by itself to produce a modified 
key value a, if desired. By modifying the relation between the foreground 

25 key value and the alpha value, the affected background region (C, and 
possibly T, in Figure 6) that the foreground subject fits into can be 
increased or reduced in size. The foreground subject remains the same size 
because the foreground suppression process has already reduced the 
outlying areas to black. Although an alpha mixer is used here, some 

30 control is retained over the background region, analogous to the control 
provided by a more complicated alpha mixer. 
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The shadow control (on/off) module (85 m Figure 4, 107 in Figure 
5) determines whether shadow information is included in the foreground 
image video. Attention here focuses on provision of shadows in the 
background so that the foreground key signal is used to determine whether 
5 a given pixel is or is not a background pixel. Although transition colors 
are not, strictly speaking, part of a background pixel, the shadow 
mechanism may be activated to provide finer detail in these transition 
regions. If the foreground key value for a given pixel is below some 
adjustable level, shadow generation is not activated for that pixel. If the 

10 luminance value is below some adjustable level, which is preferably set 
below the average luminance of the key color background, and the 
foreground key level is above its required level, shadow generation wih be 
activated for that pixel. 

The shadow generation module (87 in Figure 4, 109 in Figure 5) 

15 produces a shadow strength signal based in part on the shadow on/off 
signal. If the shadow control signal FGSh is "off, the shadow strength is 
0. The shadow signal is used to modify the alpha value. Because the 
background portions of the foreground video image have already been 
suppressed to "black" or to a darker hue, a shadow can be created in the 

20 composite image by mixing in more of the foreground color. The shadow 
strength may be added to the a value (a a', with a < a' < 1), or 
subtracted from the 1-a value (a -4 a', with a < a' < 1), where overflow 
and/or underflow are compensated for by clipping. 

The alpha mixer (45 in Figure 2, 63 in Figure 3, 125 in Figure 5) 

25 may be a standard module that receives an alpha value a, a first signal A 
and a second signal B and generates and issues a weighted sum, a* A + (1- 
a)-B, where 0 < a < 1. 

Figure 9 shows a block diagram of a general computer system 200, 
which may be used to implement various hardware components of the 

30 invention, such as a client an applications server and a database 

management system. The computer system 200 includes a bus 208 or other 
communication mechanism for communicating information and a 
processor 210, coupled with the bus 208, for processing information. The 
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computer system 200 also includes a main memory 212, such as a random 
access memory (RAM) or other dynamic storage device, coupled to the 
bus 208, for storing information and instructions to be executed by the 
processor 210. The main memory 212 also may be used for storing 
5 temporary variables or other intermediate information during execution 
of instructions by the processor 210. The computer system 200 further 
optionally includes read only memory (ROM) 214 or other static storage 
device, coupled to the bus 208, for storing static information and 
instructions for the processor 210. A storage device 216, such as a 

10 magnetic disk or optical disk, is provided and is coupled to the bus 208 for 
storing mformation and instructions. 

The computer system 200 may also be coupled through the bus to a 
display 218, such as a cathode ray tube (CRT), for displaying information 
to a computer user. An input device 220, including alphanumeric and 

15 other keys, is coupled to the bus for commimicating information and 
commands to the processor 210. Another type of user input device is a 
cursor control 222, such as a mouse, a trackball or cursor direction keys 
for communicating direction information and command selections to the 
processor 210 and for controlling cursor movement on the display 218. 

20 This input device typically has one degree of freedom in each of two axes, 
such as X- and y-axes, that allows the device to specify locations in a plane. 

The functionality of the invention is provided by the computer 
system 200 in response to the processor 210 executing one or more 
sequences of instructions contained in main memory 212. These 

25 instructions may be read into main memory 212 from another computer- 
readable medium, such as a storage device 216. Execution of the sequences 
of instructions contained in the main memory 212 causes the processor 
210 to perform the process steps described herein. In alternative 
embodiments, hard- wired circuitry may be used in place of, or in 

30 combination with, software instructions to implement the invention. 

Embodiments of the invention are not lunited to any specific combination 
of hard-wired circuitry and software. 
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The term "computer-readable medium", as used herein, refers to 
any medium that participates in providing instructions to the processor 
210 for execution. This medium may take many forms, including but not 
limited to non-volatile media, volatile media and transmission media. Non- 
5 volatile media includes, for example, optical and magnetic disks, such as 
the storage disks 216. Volatile media includes dynamic memory 212, 
Transmission media includes coaxial cables, copper wire and fiber optics 
and includes the wires that are part of the bus 208. Transmission media 
can also take the form of acoustic or electromagnetic waves, such as those 

10 generated during radiowave, infrared and optical data communications. 

Common forms of computer-readable media include, for example, a 
floppy disk, a flexible disk, a hard disk, magnetic tape or any other 
magnetic medium, a CD-ROM, any other optical medium, punchcards, 
papertape, any other physical medium with patterns of holes or apertures, 

15 a RAM, a ROM, a PROM, an EPROM, a Flash-EPROM, any other 

memory chip or cartridge, a carrier wave as described hereinafter, or any 
other medium from which a computer can be read. 

Various forms of computer-readable media may be involved in 
carrying out one or more sequences of one or more instructions to the 

20 processor 210 for execution. For example, the instructions may initially be 
carried on a magnetic disk of a remote computer. The remote computer 
can load the instructions into its dynamic memory and send the 
instructions over a telephone, using a modem. A modem local to the 
computer system 200 can receive data over a telephone line and use 

25 infrared transmitter to convert and transmit the data to the an infrared 

detector connected to the computer system bus. The bus wiE carry the data 
to the main memory 212, from which the processor receives and executes 
the instructions. Optionally, the instructions receive by the main memory 
212 can be stored on the storage device 216, either before or after 

30 execution by the processor 210. 

The computer system 200 also includes a communications interface 
224, coupled to the bus 208, which provides two-way data communication 
coupling to a network Imk 226 that is connected to a local area network 
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(LAN) or to a wide area network (WAN). For example, the 
communications interface 224 may be an integrated services digital 
network (ISDN) card or a modem to provide a data communication 
cormection to a corresponding type of telephone line. As another example, 
the communications interface 224 may be a local area network card to 
provide a data communication connection to a compatible LAN. Wireless 
links may also be implemented. In any such implementation, the 
communications interface 224 sends and receives electrical, 
electromagnetic or optical signals that carry digital data streams 
representing various types of information. 

The network link 226 typically provides data communication 
through one or more networks to other data devices. For example, the 
data link 226 may provide a connection through an LAN 228 to a host 
computer 230 or to data equipment operated by an Intemet Service 
Provider (ISP) 232. The ISP, in tum, provides data communication 
services through the world wide packet data communication network, now 
commonly known as the "Intemet" 234, served by one or more servers 
236. The LAN 228 and the Intemet 234 both use electrical, 
electromagnetic and/or optical signals to carry the digital data streams. 
The signals carried by these network, the signals carried on the network 
link 226 and the signals carried on the communications interface 224, are 
examples of carrier waves that transport the information. 



What is claimed is: 
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1. Apparatus for formation of a composite video image, the 
apparatus comprising: 

a foreground image suppression module that receives a foreground 
video image signal and, for at least one selected pixel in the foreground 
image, provides a foreground suppression key, indicating at least one key 
color that is to be suppressed for the selected pixel, and provides a 
foreground a-key, indicating a relative weight a that is to be applied to 
the selected pixel value after application of the foreground suppression 
key, where a is a selected real number lying in a range 0 < a < 1; 

a foreground suppression module that receives the foreground 
suppression key and the foreground image signal and generates a 
suppressed foreground image signal SuFG, in which the at least one key 
color is suppressed, and multiplies the suppressed foreground image signal 
by a, for the selected pixel, to form a first signal component a*SuFG; 

a background image module that receives a background video image 
signal BG and multiplies the background image signal by 1-a' for the 
selected pixel, where a' is a selected real number lying in a range 0 < oc' < 
1; and 

a sum module that forms a sum signal a'SuFG + (l-a')*BG for the 
selected pixel, 

2. The apparatus of claim 1, wherein said selected numbers a and a' 
are chosen to be equal. 

3. The apparatus of claim 1, wherein at least one of said foreground 
image suppression module and said foreground suppression module 
determines a minimum value oCmin ^^^^ ^ value, determines a 
maximum value a^iax said a value, with 0 < ajnin ^ « ^ oCj^iax - 1' 
and redefines said a-key value to provide a modified value, a'" = al or 
a'" = a2, according to at least one of the following constraints: 
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al = f(a) = 0.5 {la - a^inl - !« - amax' + («min + «max)}' 
a2 = g(al) = (al - ajnin)/(oCmax " «min)- 

4. Apparatus for formation of a composite video image, the 
apparatus comprising: 

a foreground image suppression module that receives a foreground 
video image signal and, for at least one selected pixel in the foreground 
image, provides a foreground suppression key, indicating at least one key 
color that is to be suppressed for the selected pixel, provides a foreground 
a-key, indicating a relative weight a that is to be applied to the selected 
pixel value after application of the foreground suppression key, where a is 
a selected real number lying in a range 0 < a < 1, and generates a 
suppressed foregroimd image signal SuFG, in which the at least one key 
color is suppressed; and 

an alpha mixer that receives the suppressed foreground image signal 
SuFG, receives a background video image signal BG, provides a selected 
relative weight a' that is to be applied to the selected pixel value for the 
background image, and forms a sum image signal a*SuFG + (l-a')*BG 
for the selected pixel, where a' is a selected real number lying in a range 
0 < a' < 1. 

5. The apparatus of claim 4, wherein said selected numbers a and a' 
are chosen to be equal. 

6. The apparatus of claim 4, wherein said foreground image 
suppression module determines a minimum value ajnln for said a-key 
value, determines a maximum value Oj^^^x ^^^^ ^ value, with 0 < (Xj^iji 
< a < CLj^six - ^' redefines said a value to provide a modified value, 
a'" = al or a'" = a2, according to at least one of the following 
constraints: 

al = f(a) = 0.5 {la - a^^^^l - la - amax" + («min + «max)}' 
a2 = g(al) = (al - ajnin)/(«max " «min)- 
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7. Apparatus for formation of a composite video image, the 
apparatus comprising: 

a shadow control module that receives a foreground video image 
signal and generates a shadow key that identifies at least one selected pixel 
for which foreground shadowing is activated; 

a shadow generation module that receives the foreground image 
signal and the shadow key and generates and issues a foreground shadow 
signal FGSh, in which each selected pixel appears in a shadow format; and 

a shadowing module that receives the foreground shadow signal and 
a background video image signal BG and generates a modified background 
image signal, in which a foreground shadow is impressed on each selected 
pixel of the background image. 

8. The apparatus of claim 7, wherein said shadowing module forms 
a sum signal j3-FGSh + (1-|3')*BG, as said modified background image 
signal, where P and p' are selected real numbers lying in a range [0,1]. 

9. The apparatus of claim 8, wherein said selected numbers j3 and (3' 
are chosen to be equal. 

10. Apparatus for formation of a composite video image, the 
apparatus comprising: 

a foreground image suppression/shadow module that receives a 
foreground video image signal and, for at least a first selected pixel in the 
foreground image, provides a foreground suppression key, indicating at 
least one key color that is to be suppressed for the selected pixel, provides 
a foreground a-key, indicating a relative weight oc that is to be applied to 
the selected pixel value after application of the foreground suppression 
key, where ot is a selected real number lying in a range 0 < a < 1, and 
generates a suppressed foreground image signal SuFG, in which the at 
least one key color is suppressed; 
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a shadow control module that receives the foreground image signal 
and generates a shadow key that identifies at least a second selected pixel 
for which foreground shadowing is activated; 

a shadow generation module that receives the foreground image 
signal, the shadow key and a background video image signal and generates 
and issues a foreground shadow signal and a modified background image 
signal, MBG, in which each selected pixel includes an effect of shadowing 
of a background image by a foreground image; and 

an alpha mixer that receives the suppressed foreground image signal 
SuFG, receives the foreground shadow key, receives the foreground 
shadow signal, receives the modified background image signal MBG, 
provides a first relative weight a and a second relative weight a', where 
each of a and a' is a selected real number lying in a range [0,1], that are 
to be applied to the selected pixel values for the suppressed foregroimd 
image signal and for the background image signal, and forms a sum image 
signal a*SuFG + (l-a')*MBG for the first and second selected pixels in 
which a foreground shadow is impressed on the second selected pixel of 
the background image. 

11. The apparatus of claim 10, wherein said selected numbers a and 
a' are chosen to be equal. 

12. The apparatus of claim 10, wherein said first selected pixel and 
said second selected pixel are chosen to be different pixels. 

13. The apparatus of claim 10, wherein at least one of said 
foreground image suppression module and said alpha mixer determines a 
minimum value a^]^ for said a value, determines a maximum value 
^max ^^i^ ^ value, with 0 < aj^jn ^ ot < Wmax - 1' redefines said 
a value to provide a modified value, a'" = al or a'" = a2, according to 
at least one of the following constraints: 

al = f(a) = 0.5 {la - a^i^\ - \a - a^^^l + (a^nin + «max)}' 
a2 = g(al) = (al - (Xmin)/(«max " «min)' 
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14. A method for formation of a composite video image, the method 
comprising: 

receiving a foreground video image signal and, for at least one 
selected pixel in the foreground image, providing a foreground 
suppression key, indicating at least one key color that is to be suppressed 
for the selected pixel, and providing a foreground a-key, indicating a 
relative weight a that is to be apphed to the selected pixel value after 
application of the foreground suppression key, where a is a selected real 
number lying in a range 0 < a < 1; 

receiving the foreground suppression key and the foreground image 
signal and generating a suppressed foreground image signal SuFG, in 
which the at least one key color is suppressed, and multiplying the 
suppressed foreground image signal by a, for the selected pixel, to form a 
first signal component a-SuFG; 

receiving a background video image signal and multiplying the 
background image signal BG by 1-a' for the selected pixel, where a' is a 
selected real number lying in a range 0 < a' < 1; and 

forming a sum signal, a-SuFG + (l-a')'BG, for the selected pixel. 

15. The method of claim 14, further comprising choosing said 
selected numbers a and a' to be equal. 

16. The method of claim 14, further comprising providing a 
minimum value otj^in for said a value, providing a maximum value ttj^^x 
for said a value, with 0 < ajjiin - ^ ^max - 1» redefining said a 
value to provide a modified value, a'" = al or a'" = a2, according to at 
least one of the following constraints: 

al = f(a) = 0.5 {la - ajnini - '« - a^^^l + (a^in + ocmax))' 
a2 = g(al) = (al - ajnin)/(0Cmax " «min)- 
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17. A method for formation of a composite video image, the method 
comprising: 

receiving a foreground video image signal and, for at least one 
selected pixel in the foreground image, providing a foreground 
suppression key, indicating at least one key color that is to be suppressed 
for the selected pixel, providing a foreground a-key, indicating a relative 
weight a that is to be applied to the selected pixel value after application 
of the foreground suppression key, where a is a selected real number 
lying in a range 0 < a < 1, and generating a suppressed foreground image 
signal SuFG, in which the at least one key color is suppressed; and 

receiving the suppressed foreground image signal SuFG, receiving a 
background video image signal, providing a relative weight ct' that is to be 
applied to the selected pixel value for the background image, and forming 
a sum image signal a-SFG + (l-a')*BG for the selected pixel, where a' is 
a selected real number lying in a range 0 < a' < 1. 

18. The method of claim 17, further comprising choosing said 
selected numbers a and a' to be equal. 

19. The method of claim 17, further comprising providing a 
minimum value C(^{ji for said a value, providing a maximum value cl^^^ 
for said a value, with 0 < amin - oc ^ ocmax - 1' redefining said a 
value to provide a modified value, a'" = al or a'" = a2, according to at 
least one of the following constraints: 

al = f(a) = 0.5 {la - ocmin' " '« " «max" + («min + «max)}' 
a2 = g((xl) = (al - amin)/(«max " «min)- 

20. A method for formation of a composite video image, the method 
comprising: 

receiving a foreground video image signal and generating a shadow 
key that identifies at least one selected pixel for which foreground 
shadowing is activated; 
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receiving the foreground image signal and the shadow key and 
generating and issuing a foreground shadow signal, FGSh, in which each 
selected pixel appears in a shadow format; and 

receiving the foreground shadow signal and a background video 
image signal and generating a modified background image signal, MBG, in 
which a foreground shadow is impressed on each selected pixel of the 
background image. 

21. The method of claim 20, wherein said process of generating said 
modified background image signal comprises forming a sum signal 
p'FGSh + (l-p')'BG, where p and p' are selected real numbers lying in a 
range [0,1]. 

22. The method of claim 21, further comprising choosing said 
selected numbers p and P' to be equal. 

23. A method for formation of a composite video image, the method 
comprising: 

receiving a foreground video image signal and, for at least a first 
selected pixel in the foreground image, providing a foreground 
suppression key, indicating at least one key color that is to be suppressed 
for the selected pixel, providing a foreground a-key, indicating a relative 
weight a that is to be applied to the selected pixel value after application 
of the foreground suppression key, where a is a selected real number 
lying in a range 0 < a < 1, and generating a suppressed foreground image 
signal SuFG, in which the at least one key color is suppressed; 

receiving the foreground image signal and generating a shadow key 
that identifies at least a second selected pixel for which foreground 
shadowing is activated; 

receiving the foreground image signal and the shadow key and a 
background video image signal and generating and issuing a foreground 
shadow signal and a modified background image signal, MBG, in which 
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each selected pixel includes an effect of shadowing of a background image 
by a foreground image; and 

receiving the suppressed foreground image signal SuFG, receiving 
the foreground shadow key, receiving the foreground shadow signal, 
receiving the modified background image signal MBG, providing a first 
relative weight a and a second relative weight a', where each of a and a' 
is a selected real number lying in a range [0,1], that are to be applied to 
the selected pixel values for the suppressed foreground image signal and 
for the modified background image signal, and forming a sum image 
signal a'SuFG + (l-a')*MBG for the first and second selected pixels in 
which a foreground shadow is impressed on the second selected pixel of 
the background image. 

24. The method of claim 23, further comprising choosing said first 
selected pixel and said second selected pixel to be the same pixel. 

25. The method of claim 23, further comprising choosing said first 
selected pixel and said second selected pixel to be different pixels. 

26. The method of claim 23, further comprising providing a 
minimum value a^^j^ for said a value, providing a maximum value a^iax 
for said a value, with 0 < a^\^ <a< Wmax - 1' redefining said a 
value to provide a modified value, a'" = al or a'" = a2, according to at 
least one of the following constraints: 

al = f(a) = 0.5 {la - ajnin" " "« " «max" + («min + «max)}' 
a2 = g(al) = (al - cXminVC^max " «min)- 

27. An article of manufacture comprising: 

a computer usable medium having computer readable code means 
embodied therein for producing a composite video image including 
portions of at least one foreground image and of at least one background 
image; 
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computer readable program code means for receiving a foreground 
video image signal and, for at least a first selected pixel in the foreground 
image, providing a foreground suppression key, indicating at least one key 
color that is to be suppressed for the selected pixel, providing a 
foreground a-key, indicating a relative weight a that is to be applied to 
the selected pixel value after application of the foreground suppression 
key, where a is a selected real number lying in a range 0 < a < 1, and 
generating a suppressed foreground image signal SuFG, in which the at 
least one key color is suppressed; 

computer readable program code means for receiving the 
foreground suppression key and the foreground image signal and 
generating a suppressed foreground image signal SuFG, in which the at 
least one key color is suppressed, and multiplying the suppressed 
foregroimd image signal by a, for the selected pixel, to form a first signal 
component oc-SuFG; 

computer readable program code means for receiving a background 
video image signal and multiplying the background image signal BG by 
1-a' for the selected pixel, where a' is a selected real number lying in a 
range [0,1]; and 

computer readable program code means for forming a sum signal, 
a-SuFG + (l-a')-BG, for the selected pixel. 

28. The article of claim 27, further comprising computer readable 
program code means for providing a minimum value CLjxnn ^^^^ ^ 
value, providing a maximum value cXj^ax ^^^^ ^ value, with 0 < a^]^ 
<a< ocmax - 1' redefining said a value to provide a modified value, 
a'" = al or a'" = a2, according to at least one of the following 
constraints: 

al = f(a) = 0.5 {la - a^i^\ - la - a^^^\ + {a^^^ + a^ax)}' 
a2 = g(al) = (al - ajnin)/(0Cmax " ^min)- 
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29. An article of manufacture comprising: 

a computer usable medium having computer readable code means 
embodied therein for producing a composite video image including 
portions of at least one foreground image and of at least one background 
image; 

computer readable program code means for receiving a foreground 
video image signal and, for at least one selected pixel in the foreground 
image, providing a foreground suppression key, indicating at least one key 
color that is to be suppressed for the selected pixel, providing a 
foreground a-key, indicating a relative weight a that is to be applied to 
the selected pixel value after application of the foreground suppression 
key, where a is a selected real number lying in a range 0 < oc < 1, and 
generating a suppressed foreground image signal SuFG, in which the at 
least one key color is suppressed; and 

computer readable program code means for receiving the 
suppressed foreground image signal SuFG, providing a relative weight a' 
that is to be applied to the selected pixel value for the suppressed 
foreground image signal, receiving a background video image signal, and 
forming a sum image signal wSuFG + (l-a')*BG for the selected pixel, 
where a' is a selected real number lying in a range [0,1]. 

30. The article of claim 29, further comprising computer readable 
program code means for providing a minimum value oCjjjjjj for said a 
value, providing a maximum value ajjj^x ^^r said a value, with 0 < amin 
< a < oCjjjax - 1' redefining said a value to provide a modified value, 
a'" = al or a'" = a2, according to at least one of the following 
constraints: 

al = f(a) = 0.5 {la - a^inl - loc - a^ax' + (^min + «max)}' 
a2 = g(al) = (al - ajnin)/(0Cmax " ^min)- 

3 1 . An article of manufacture comprising: 

a computer usable medium having computer readable code means 
embodied therein for producing a composite video image including 
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portions of at least one foreground image and of at least one background 
image; 

computer readable program code means for receiving a foreground 
video image signal FG and generates a shadow key that identifies at least 
one selected pixel for which foreground shadowing is activated; 

computer readable program code means for receiving the 
foreground image signal and the shadow key and generating and issues a 
foreground shadow signal FGSh, in which each selected pixel appears in a 
shadow format; and 

computer readable program code means for receiving the 
foreground shadow signal and a background video image signal BG and 
generating a modified background image signal, in which a foreground 
shadow is impressed on each selected pixel of the background image. 

32. The article of claim 31, further comprising computer readable 
program code means for forming a sum signal p*FGSh + (l-p')*BG as 
said modified background image signal, where p and p' are selected real 
numbers lying in a range [0,1]. 

33. An article of manufacture comprising: 

a computer usable medium having computer readable code means 
embodied therein for producing a composite video image including 
portions of at least one foreground image and of at least one background 
image; 

computer readable program code means for receiving a foreground 
video image signal and, for at least a first selected pixel in the foreground 
image, providing a foreground suppression key, indicating at least one key 
color that is to be suppressed for the selected pixel, providing a 
foreground a-key, indicating a relative weight a that is to be applied to 
the selected pixel value after application of the foreground suppression 
key, where a is a selected real number lying in a range 0 < a < 1 , and 
generating a suppressed foreground image signal SuFG, in which the at 
least one key color is suppressed; 
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computer readable program code means for receiving the 
foreground image signal and generating a shadow key that identifies at 
least a second selected pixel for which foregroimd shadowiag is activated; 

computer readable program code means for receiving the 
foreground image signal and the shadow key and generating and issuing a 
foreground shadow signal, in which each selected pixel appears in a 
shadow format; and 

computer readable program code means for receiving the 
suppressed foreground image signal SuFG, receiving the foreground 
shadow key, receiving the foreground shadow signal, receiving a 
background video image signal BG, providing a first relative weight a and 
a second relative weight a', where each of a and a' is a selected real 
number lying in a range [0,1], that are to be applied to the selected pixel 
values for the suppressed foreground image signal and for the background 
image signal, and forming a sum image signal a-SuFG + (l-a')*BG for 
the first and second selected pixels in which a foreground shadow is 
impressed on the second selected pixel of the background image. 

34. The article of claim 33, further comprising computer readable 
program code means for choosing said first selected pixel and said second 
selected pixel to be the same pixel. 

35. The article of claim 33, further comprising computer readable 
program code means for choosing said first selected pixel and said second 
selected pixel to be different pixels. 

36. The article of claim 33, further comprising computer readable 
program code means for providing a minimum value oCjjjjj^ for said a 
value, providing a maximum value Wmax ^^^^ ^ value, with 0 < oCj^jj^ 
< a < oCj^ax - 1' redefining said a value to provide a modified value, 
a'" = al or a'" = a2, according to at least one of the following 
constraints: 
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f(a) = 0.5 {la - ajnini - '« - ctmax' + (Omin + Wmax))' 
g(al) = (al - aniin)/(amax " «min)- 
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Abstract of the Invention 

A system for forming composite video images from one or more 
foreground images and one or more background images. In one 
embodiment, s sum of a suppressed foreground image signal with weight 
a , with one or more selected foreground colors suppressed, and a 
background image signal with weight 1-a' (0<a'<l) is formed, where a 
and a' may vary from pixel to pixel and with time. In another 
embodiment, a shadow from a selected foreground image is impressed on 
selected pixels of a background image. In another embodiment, 
foreground suppression and shadowing are combined, optionally by 
retrofitting, using an existing a-mixer or a newly constructed a-mixer. 
Provision of a chroma key map allows a foreground image shadow to be 
prescribed pixel by pixel, iacluding a transition region in which the 
shadowed image slowly disappears. 
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months prior to the filing date of this application, if any, are identified below. 

CHECK APPROPRIATE BOX 
X No earlier-filed foreign applications. 

Requirement information as to foreign applications filed prior to filing date of this 

application 

is on page attached hereto and made a part hereof. 
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POWER OF ATTORNEY : 

As a named inventor, I hereby appoint the following attomey(s) and/or agent(s) to prosecute this 
apphcation and transact all business in the Patent and Trademark Office connected therewith: 
Bobby K. Truong, Reg. No. 37,499, John F. Schipper, Reg. No. 26,994, Stanley N. Protigal, Reg. No. 
28,657, Richard E. Bee, Reg. No. 18,005 and George M. Steres, Reg No. 36,690 of the Law Offices of 
Sabath & Truong. 



SEND CORRESPONDENCE TO 

SABATH & TRUONG 

1 1 1 North Market Street, Suite 815 

San Jose, CA 95113 

Attn: John F. Schipper 



DIRECT TELEPHONE CALLS TO 

John F. Schipper 
Tel: (408) 293-9934 
Fax: (408) 293-2183 



FULL NAME OF 
INVENTOR 1 


LAST NAME 

AGARWALA 


FIRST NAME 

VINAY 


MIDDLE NAME 


RESIDENCE 
& 

CITIZENSHIP 


CITY 

SAN JOSE 


STATE OR FOREIGN 
COUNTRY 

CA 


COUNTRY OF 
CITIZENSHIP 
U.S.A. 


POST 

OFFICE 

ADDRESS 


POST OFFICE 
ADDRESS 

1333 PHELPS AVE 


CITY 

SAN JOSE 


STATE OR 
COUNTRY 
USA 


ZIP CODE 
95117 


FULL NAME OF 
INVENTOR 2 


LAST NAME 

TSE 


FIRST NAME 
CLEMENT 


MIDDLE NAME 


RESIDENCE 
& 

CITIZENSHIP 


CITY 

FREMONT 


STATE OR FOREIGN 

COUNTRY 

CA 


COUNTRY OF 
CITIZENSHIP 
U.S.A. 


POST 

OFFICE 

ADDRESS 


POST OFFICE 
ADDRESS 
681 UPPER 
VINTERS CIRCLE 


CITY 

FREMONT 


STATE OR 
COUNTRY 
USA 


ZIP CODE 

94539 
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I further declare that all statements made herein of my own knowledge are true and that all statements 
made on information and belief are beheved to be true; and further that these statements were made 
with the knowledge that false statements and the like so made are punishable by fme or imprisonment, 
or both, under section 1001 of Title 18 of the United States Code, and that such willful false statements 
may jeopardize the validity of the apphcation or any patent issuing thereon. 



NAME (#1) 

VINAY AGARWALA 


SIGNATURE , 


DATE 




NAME (#2) 
CLEMENT TSE 


SIGNATURE 


DATE 
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Section 1 .56 Duty to Disclose Information Material to Patentability, 
(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when,, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in dealing 
with the Office, which includes a duty to disclose to the Office all information known to that individual to be 
material to patentability of a claim that is cancelled or withdrawn from consideration need not be submitted if 
the information is not material to the patentablity of any claim remaining under consideration in the 
application. There is no duty to submit information which is not material to the patentability of any existing 
claim. The duty to disclose all information known to be material to patentabihty is deemed to be satisfied if all 
information known to be material to patentability of any claim issued in a patent was cited by the Office or 
submitted to the Office in the manner prescribed by Sections 1 .97(b)-(d) and 1 .98. However, no patent will be 
granted on an application in connection with which fraud on the Office was practiced or attempted or the duty 
of disclosure was violated through bad faith or intentional misconduct. The Office encourages applications to 
carefully examine: 

(1) prior art cited in search report os a foreign patent office in a counterpart appUcation, 

and 

(2) the closest information over which individuals associated with the filing or 
prosecution of a patent application believe any pending claim patentably defines, to make sure that any 
material information contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record of being made of record in the application, and 

(1) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the application takes in: 

(i) opposing an argument of unpatentability reUed on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim its 
broadest reasonable construction consistent with the specification, and before any considerations given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent appUcation within the meaning 
of this section are: 

( 1 ) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecutionn of 
the application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent or inventor. 



Page 4 of 4 



